Image processing method and apparatus therefor

ABSTRACT

A system is provided for obtaining parameters about the position and orientation of a camera, where the camera has a fixed position and orientation, by using the image coordinates and world coordinates of markers included in an image photographed by the camera. This system uses a live-image display mode for successively obtaining images input to the camera and a still-image display mode for obtaining an image input to the camera at predetermined time. The markers can be extracted by using the image obtained in either mode. The system can select either automatic-extraction mode or manual-extraction mode for extracting the marker in live images, so that the markers can be extracted with a high degree of accuracy under all conditions to obtain the position and orientation parameters of the camera.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to an image processing method andan apparatus therefore for performing calibration of the position andorientation of an image pickup device.

[0003] 2. Description of the Related Art

[0004] Conventional methods for calibrating the position and orientationof a camera that is fixed in a predetermined space are well known.According to one method, a plurality of markers with known coordinatesin a three dimensional space (hereinafter referred to as “worldcoordinates”) are photographed by the camera and the position andorientation of the camera is calculated. The position and orientationsatisfies the relationship between the world coordinates of the markersand the coordinates of the markers in the photographed image. That is,this calculation solves the perspective n-point (PnP) problem.

[0005] In this method, the markers are extracted from the photographedimage and are identified using a marker extraction process. This markerextraction process can be performed either automatically, manually or bya combination of both.

[0006] Where the marker extraction process is performed manually by theoperator, the photographed image is displayed as a still image on adisplay. Then, the operator clicks the mouse button on the positions ofthe markers on the displayed image to input image coordinates of themarkers.

[0007] Where the marker extraction process is performed automatically,image processing is performed for the still image, which is photographedby the camera, by setting and adjusting marker extraction parameters fordetermining the threshold values of colors for extraction and the sizesof the markers.

[0008] However, marker extraction can be impacted by image brightness.When the contrast level of the image is low in accordance with thebrightness of a photographed space and a photographing direction, itbecomes difficult to identify the markers on the photographed image. Itshould be noted that appropriate lighting can be performed duringphotographing. Where the marker extraction process is performedautomatically, the marker extraction parameters need to be adjusted.Therefore, much trial and error is required for setting predeterminedsurroundings for performing good marker extraction process.

[0009] Further yet, a known method for automatically extracting markerscaptures a still image in order to extract the markers. If the markerscannot be properly extracted, the surrounding thereof is changed andphotographed again. Then, the marker extraction is repeated. Thus, themarker extraction process needs to be performed repetitively. Where themarker extraction process is manually performed by the operator, aphotographed image must be repetitively input so as to make informationrequired for the marker extraction visible for every marker.

SUMMARY OF THE INVENTION

[0010] According to one aspect of the present invention, at least onemode is selected from among a plurality of obtain modes including afirst obtain mode for obtaining images picked up by an image pickupdevice, as moving images, and a second obtain mode for obtaining atleast one image picked up by the image pickup device, as a still image.Then, an image picked up by the image pickup device in the selectedobtain mode is obtained. Further, a marker included in the obtainedimage is extracted, and information about the position and orientationof the image pickup device is derived by using the image coordinates ofthe extracted marker.

[0011] Further features and advantages of the present invention willbecome apparent from the following description of the embodiments (withreference to the attached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1 illustrates a UI (User Interface) for performing markerextraction according to a first embodiment of the present invention.

[0013]FIG. 2 is a block diagram illustrating the schematic configurationof a system according to the first embodiment.

[0014]FIG. 3 is a flowchart illustrating processing procedures accordingto the first embodiment, where the processing procedures calibrate theposition and orientation of a camera.

[0015]FIG. 4 illustrates a UI for performing the marker extractionprocess, where live-image display mode and manual-extraction processmode are used in combination, according to an embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

[0016] Embodiments of the present invention will now be described indetail with reference to the attached drawings.

FIRST EMBODIMENT

[0017] A first embodiment of the present invention provides a system forcalibrating the position and orientation of a fixed camera. The cameraphotographs a marker with known world coordinates and the calibrationsystem calculates a position and orientation of the camera (cameraexternal parameters), where the position and orientation satisfies therelationship between the world coordinates of the marker and the imagecoordinates thereof.

[0018] The following factors are used in this embodiment:

[0019] 1. the world coordinates of at least three markers that are notin line with one another;

[0020] 2. the image coordinates of the markers on an image photographedby the camera; and

[0021] 3. the initial values (roughly estimated values) of the positionand orientation of the camera in a world-coordinate system.

[0022] The theoretical value of the image coordinates of each of themarkers is calculated based on the world coordinates of the marker andthe position and orientation of the camera. Then, a position andorientation of the camera for minimizing the sum of differences betweenthe theoretical values of the markers and the measurement values thereofis calculated. This process is referred to as camera calibration.

[0023] The data given under 1. and the data given under 3. is preparedbefore starting the calibration process, as known information. At leastthree markers (landmarks or feature points) with known world coordinatesmust be determined in a real space photographed by the camera. Thesemarkers have certain features, which can be detected and identified onan image photographed by the camera. For example, markers featurecolored circles, each of which has a different color. In the worldcoordinate system, a point is selected to be an origin from which the X,Y, and Z axes extend. Next, the distance between the origin point andeach of the three markers along each of the X, Y, and Z directions ismeasured. Although the data given under 3., that is, the initial valuesof the position and orientation of the camera is often unnecessary, itfacilitates obtaining correct solutions to the above-describedcalculation. Therefore, the initial values are roughly estimated.

[0024] The data given under 2. calculates the position and orientationof the camera. This data is obtained when markers are extracted from animage photographed by the camera. In this embodiment, the markerextraction process can be automatically performed by using markerextraction parameters, described below. Further, the operator canmanually input the marker positions.

[0025] According to this embodiment, a user interface (UI) 10 for themarker extraction process is provided, as shown in FIG. 1. This UI 10has an image display unit 20 for displaying both the image and theresult of the marker extraction process. In this drawing, each of thepositions of extracted markers is indicated by a symbol × and each ofthe identified markers is enclosed by a box.

[0026] This UI 10 has the function of selecting the image-display mode.The operator can switch between live-image display mode and still-imagedisplay mode by pressing a live/still button 30. When this apparatus isstarted, the live-image mode is selected. In this mode, theimage-display unit 20 displays the latest live images that aresuccessively photographed by the camera. These live images become thesubject of the marker extraction process. Hereinafter, “subject image,”is one that is displayed on the image-display unit 20 and that is thesubject of the marker extraction process. In the still-image displaymode, a previously read image becomes the subject image. Morespecifically, in the still-image display mode, an image that is input atthe time when the display mode is switched from the live-image displaymode to the still-image display mode through the live/still button 30becomes the subject image.

[0027] If the level of noise detected in the marker extraction processis high, the operator can select the live-image display mode by usingthe above-described image-display-mode switching function. In this case,since the operator can change the brightness of images while observingthe images changing in real time, suitable brightness for performing theautomatic marker extraction process can be obtained to extract themarkers. Further, where the operator extracts markers manually and wherethe entire markers are not visible at the same time, the operator cansuccessively extract markers shown on live images. However, if it isdifficult to keep photographing markers due to constant movement ofpeople, for example, the still-image display mode can be selected afterthe initial photograph of the markers. Subsequently, the operator canperform the marker extraction process without being affected by suchconstant movement of people.

[0028] This UI 10 also functions to select the marker-extraction processmode. This marker-extraction process mode is provided for determiningwhether markers on a subject image are extracted manually orautomatically. Therefore, the marker-extraction process mode includesautomatic-extraction process mode and manual-extraction process mode. Amarker-extraction switch button 40 is used for switching back and forthbetween the automatic-extraction process mode and the manual-extractionprocess mode. In the automatic-extraction process mode, the subjectimage is subjected to image processing according to marker-extractionparameters, described below, so that the markers are extractedautomatically. In the manual-extraction process mode, the operator canmanually input the positions of the markers on the subject image byclicking the mouse button on the marker positions on the subject imagedisplayed on the image-display unit 20. Further, the operator can selectany of the markers by clicking the mouse button on the marker on thesubject image displayed on the image-display unit 20, delete a markerextracted by mistake by using a marker delete button 50, and identifyand correct any of the markers by using a marker-identificationpull-down menu 60. In this manual-extraction process mode, the operatorcan manually correct part of markers extracted by mistake by using theresult of extraction process obtained in the automatic-extractionprocess mode. Note that the operator can extract all of the markersmanually without using the result of the automatic extraction process.

[0029] Further, the operator can use UI 10 to change the markerextraction parameters in an interactive manner through an input unit(not shown). The marker extraction parameters include parameters forspecifying a predetermined marker for extraction, where theabove-described marker-extraction process mode is theautomatic-extraction process mode. These parameters include, forexample, the threshold values of different colors including red, green,and blue, and the sizes of markers. Note that these parameters forperforming the marker extraction process should be correctly set. In theautomatic-extraction process mode, the operator can adjust the markerextraction parameters through this UI 10 with trial and error byreferring to the result of the marker extraction process.

[0030] A camera-parameter calculation execution button 70 is provided onthe UI 10. When the operator presses this button 70 after the markerextraction process is finished, a position and orientation of the camerais calculated, where the position and orientation satisfies therelationship between the image coordinates of the extracted marker andthe world coordinates thereof. This process is referred to as cameracalibration.

[0031]FIG. 2 is a block diagram schematically illustrating a calibrationsystem 1000 of the present embodiment.

[0032] In FIG. 2, a camera to be calibrated is connected to thiscalibration system 1000. An image pickup device 100 is also connected tothe calibration system 1000. This image pickup device 100 is fixed to apredetermined position in a subject space, so as to take a view of thesubject space and markers determined therein.

[0033] A picked-up-image capturing unit 200, which inputs an imagesignal photographed by the fixed image pickup device 100, converts theinput image signal into a predetermined format suitable for a displayunit 700 and a marker extraction unit 300, and transmits the convertedimage signal to the display unit 700 and the marker extraction unit 300on receiving a request from the marker extraction unit 300.

[0034] An operation input unit 800 is realized by various types ofinterfaces including a mouse, a keyboard, and so forth. A signal inputby the operator is input to this operation input unit 800, as data. Theimage-display mode selection function, the marker-extraction processmode selection function, and the extraction-parameter change functionused for extracting markers are achieved by this operation input unit800. The operation input unit 800 is used for inputting data, where theoperator extracts a marker manually, and inputting an instruction forexecuting a command issued for calculating the position and orientationof the camera.

[0035] The signal input from the operation input unit 800 is convertedinto a predetermined format and transmitted to a processing unittherefor. More specifically, a command for the changing mode or theextraction parameters is transmitted to the marker extraction unit 300and a command for calculating the position and orientation of the camerais transmitted to a position and orientation calculation unit 400.

[0036] The mode-changing process and the marker extraction process areperformed in the marker extraction unit 300. Upon receiving a commandfor changing the image display mode and the marker-extraction processmode from the operation input unit 800, the marker extraction unit 300changes the mode according to the input command. Further, upon receivinga command for changing the marker-extraction parameters from theoperation input unit 800, the marker extraction unit 300 changes thevalues of marker-extraction parameters, according to the input command.

[0037] In the live-image mode, the marker extraction unit 300 transmitsan instruction for transmitting a photographed image to thepicked-up-image capturing unit 200 and reads the transmittedphotographed image. Subsequently, the marker extraction unit 300 can usethe latest image as the subject image. In the still-image display mode,the marker extraction unit 300 determines that a photographed imagepreviously read at the time when the still-image display mode isselected is the subject image. The marker extraction unit 300 does notupdate this subject image until when the live-image display mode isselected.

[0038] In the automatic-extraction process mode, the marker extractionunit 300 performs image processing according to the marker informationread from a data unit 600 and the above-described marker extractionparameters, thereby extracting markers on the subject image. In thismode, whenever the subject image is updated, the markers aresuccessively extracted and the result of marker extraction is updated.

[0039] In the manual-extraction process mode, data input by the operatorand transmitted from the UI 10 is determined to be the result of themarker extraction process of the subject image. If this mode isselected, the position of specified data is used, even though thesubject image is updated.

[0040] When an input is received via the camera-parameter calculationexecution button 70, the marker extraction unit 300 transmits markerextraction information, which is the result of the marker extractionprocess, to the position and orientation calculation unit 400. If thenumber of extracted markers is less than three at the time where thecamera-parameter calculation execution button 70 is pressed, where threeis a minimum number required for performing calibration, an alarm isissued by the marker extraction unit 300. In other embodiments, thecamera-parameter calculation execution button 70 is not responsive untilthe number of extracted markers becomes three.

[0041] Upon receiving a command for calculating the position andorientation of the camera from the operation input unit 800, theposition and orientation calculation unit 400 calculates the positionand orientation of the camera. The position and orientation calculationunit 400 then calculates the theoretical values of image coordinates ofthe markers, based on the world coordinates of the markers and theinitial values of the position and orientation of the camera. Next, theunit calculates a position and orientation of a camera so as to minimizethe sum of differences between the theoretical values of the markers andthe measurement values thereof transmitted from the marker extractionunit 300, as marker information. The value of the calculated positionand orientation is transmitted to an image generation unit 500.

[0042] The information on each of the markers includes the worldcoordinates, the identifier (ID), and so forth. Such information can bemanually changed or corrected by the operator by using a setting unit(not shown). The set marker information is provided in list form in a“Marker list” area in the UI 10, as shown in FIG. 1.

[0043] Since at least three markers are required in this embodiment, analert signal is provided if the number of markers set by the settingunit is less than three.

[0044] The image generation unit 500 generates a computer graphic (CG)image for making the result of marker extraction visible, based on themarker extraction information transmitted from the marker extractionunit 300. The CG image is used for indicating the marker position by thesymbol × and enclosing each of the identified markers by a box. Afterthe position and orientation information is transmitted, the imagegeneration unit 500 generates a CG image (the X, Y, and Z axes inFIG. 1) for determining whether the transmitted information is accurate.The generated CG image data is transmitted from the image generationunit 500 to the display unit 700.

[0045] The data unit 600 controls various types of data. The dataincludes marker information, information about the position andorientation of the image pickup device 100 obtained in early stages ofthe calibration, marker extraction parameters, marker extractionalgorithms, camera-position and orientation calculation algorithms, dataon CG image generated by the image generation unit 500, and so forth.

[0046] The display unit 700 superimposes the CG image generated by theimage generation unit 500 on an image transmitted from thepicked-up-image capturing unit 200 and displays an obtained image on theimage display unit 20. The operator determines the state of extractedmarkers and the accuracy of the calculated position and orientation ofthe camera by observing the superimposed image. The operator observesthe symbol × or the box shown in FIG. 1 indicating the position ofidentified marker. Where the operator determines that the markerextraction process is completed, a command for calculating the positionand orientation of the camera is issued by using the camera-parametercalculation execution button 70. When the position and orientation ofthe camera is calculated, the operator observes the X, Y, and Z axesshown in FIG. 1. If the accuracy of the calculated position andorientation is determined to be good, the operator terminates thecalibration.

[0047]FIG. 3 is a flowchart illustrating the marker extraction processaccording to this embodiment.

[0048] The marker extraction process of this embodiment is started atstep S000. Data on initial settings is read from the data unit 600, andinitial settings are made. The live-image mode is selected as theimage-display mode and the automatic extraction mode is selected as themarker-extraction process mode. Then, default values are provided forthe marker extraction parameters. Further, data on the markers istransmitted to the marker-extraction process unit 300. The data on themarkers includes information about the identifiers, positions, colors,sizes, shapes, and so forth of the markers.

[0049] At step S100, the operation input unit 800 determines whether aninput command has been received from an operator. If an input commandhas been received, it is transmitted to the marker extraction unit 300.The marker extraction unit 300 determines the type of the command andperforms a suitable process. Where the command is issued for changingthe marker-extraction process mode, the marker extraction unit 300changes the marker extraction mode. Where the command is issued tochange the marker extraction parameters, the marker extraction unit 300changes the specified parameters. Where the command is issued tocalculate the position and orientation of the camera, the markerextraction unit 300 generates a flag for indicating that the markerextraction process is finished.

[0050] At step S200, the marker extraction unit 300 performsimage-display mode determination. If the image display mode is thestill-image mode (subject image previously read), the process advancesto step S400. Where the image display mode is the live-image displaymode and the subject image is not read, the process advances to stepS300.

[0051] At step S300, the marker extraction unit 300 requests thepicked-up-image capturing unit 200 to transmit an image. Thepicked-up-image capturing unit 200 transmits an image from the imagepickup device 100 to the marker extraction unit 300 and the display unit700.

[0052] At step S400, the marker extraction unit 300 determines the typeof marker-extraction process mode. If the automatic-extraction processmode is used, the process advances to step S411. If themanual-extraction process mode is used, the process advances to stepS421.

[0053] The processing procedures performed at steps S411 to S413, whenthe automatic-extraction process mode is selected, are described asfollows.

[0054] At step S411, markers are automatically detected from the subjectimage through image processing according to the marker-extractionparameters. The result of the marker-extraction process is transmittedto the image generation unit 500.

[0055] At step S412, the image generation unit 500 generates a CG imagefor making the marker extraction result visible. This CG image istransmitted to the display unit 700.

[0056] At step S413, the subject image and the result of themarker-extraction process are displayed on the display unit 700. Theoperator is aware of the state of marker extraction by observing thedisplay unit 700.

[0057] The processing procedures performed at steps S421 to S423, whenthe manual-extraction process mode is selected, are described asfollows.

[0058] At step S421, the image generation unit 500 generates a CG imagefor making all the results of past marker extraction visible. This CGimage is transmitted to the display unit 700.

[0059] At step S422, the subject image and all the result of the pastmarker extraction are displayed on the display unit 700. The operator isaware of the state of marker extraction by observing the display unit700.

[0060] At step S423, marker information input by the operator throughthe operation input unit 800 is maintained, as the result of themarker-extraction process. This result is transmitted to the imagegeneration unit 500.

[0061] If the marker extraction process is finished (as indicated bygeneration of a flag), the process advances to step S600. Otherwise, theprocess returns to step S100 and the marker extraction process isrepeated.

[0062] At step S600, the position and orientation calculation unit 400calculates the position and orientation of the camera. In thisembodiment, this position and orientation calculation unit 400calculates the theoretical value of the image coordinates of each of themarkers, based on the world coordinates of each of the markers givenunder 1., above, and the position and orientation of the camera givenunder 3., above, that are stored at the early stage of the calibration,as known information. Then, the position and orientation calculationunit 400 calculates a position and orientation of the camera by usingknown nonlinear iterative least minimization method for minimizing thesum of differences between the theoretical values of the markers and themeasurement values thereof.

[0063] If the operator determines that calculation of the position andorientation of the camera is failed at step S700, the process returns tostep S100 and the above-described procedures are repeated until thecalculation succeeds. Where the operator determines that the calculationsucceeded, the process is terminated.

[0064] Thus, in this embodiment, the position and orientation of thecamera is calculated, based on the differences between the theoreticalvalues of the markers and the measurement values thereof. However, theposition and orientation can be calculated according to another method,so as to satisfy the relationship between the world coordinates of themarkers and the coordinates of the picked-up images. That is, theposition and orientation can be calculated by solving a PnP problem.

[0065] Further, in this embodiment, the calibration of the cameraposition and orientation is performed by using at least one picked-upimage showing the markers with known world coordinates. However, thepresent invention can be achieved in various ways. For example, thecalibration can be performed by extracting markers or natural featurepoints shown in the picked-up image. That is, the present invention canbe used for calibration of camera parameters irrespective of whether theparameters are internal (intrinsic), external (extrinsic), or both ofthem. The camera external (extrinsic) parameters means information aboutthe position and orientation of a camera that is fixed in apredetermined space, and the camera internal (intrinsic) parametersmeans information about the lens distortion and focus distance of acamera. Further, the present invention can be used for a system forphotographing an object with markers thereon and detecting the positionand orientation of the photographed object, where the position (worldcoordinates) of the camera is known.

[0066] As has been described, at least three markers that are not inline with one another are extracted for calibrating the position andorientation of a camera.

[0067] According to this embodiment, both the live-image display modeand the still-image display mode, as well as the automatic-extractionprocess mode and the manual-extraction process mode for markerextraction are used based on various conditions. Although all necessarymarkers are not displayed, marker extraction may still be performed withrelative ease.

[0068] For example, if the necessary markers become alternately visibleand invisible, so that it becomes difficult to simultaneously photographall of the necessary markers, the live-image display mode and themanual-extraction process mode are used in combination. Thus, if thenecessary markers are visible in a live image, the operator manuallyspecifies the markers to extract them. Since live images aresuccessively obtained until the operator finishes specifying all thenecessary markers, all the necessary markers can be used for thecalibration.

[0069]FIGS. 4A and 4B illustrate use of the live-image display mode andthe manual-extraction process mode in combination. In FIG. 4A, anoperator cannot specify the marker Red_LEFT in the lower-left part of atable because it is hidden by a person 80. Therefore, markers other thanthe marker Red_LEFT can be specified. However, where the person 80 movesto the middle part of the image, as shown in FIG. 4B, the operator canspecify the left marker Red_LEFT. In this drawing, two markers that werespecified by the operator are hidden by the person 80. However, thesemarkers were already specified by the operator in the past, as shown inFIG. 4A. Therefore, a graphic image is displayed on the screen forindicating that these markers had already been specified and theinformation about these specified markers. Thus, the operator is awareof all the markers specified in the past, even though the markers arehidden.

[0070] Since the present invention is used for a case where the positionof a camera is fixed, the position of each of markers displayed on animage is fixed, even though the operator extracts the markers on animage photographed at different time, as in the case where all themarkers displayed on a single image are photographed.

[0071] If the markers are photographed in good condition once, where itis difficult to continue photographing markers in good condition, theoperator selects the still-image display mode and captures a stillimage, thereby extracting necessary markers. The user can easily extractthe markers by using the automatic-extraction process mode. Further, asdescribed above, after the markers are extracted through the automaticextraction process, the operator can select the manual-extractionprocess mode and adjust the extracted markers. Subsequently, theaccuracy of the marker extraction increases.

[0072] The marker extraction is significantly affected by environmentalconditions, such as illumination conditions of the real space.Therefore, adjustment of either the environmental conditions or themarker extraction conditions may be performed. For performing thisadjustment, the live-image display mode and the automatic-extractionprocess mode may be used in combination, so as to obtain the adjustmentresult in real time.

[0073] Thus, according to this embodiment, the position and orientationof a fixed camera can be quickly and more accurately calibrated withoutdifficulty.

[0074] In the above-described embodiment, the still-image display modeand the live-image display mode can be used, as the display mode.However, other display modes such as intermittent display mode may beused, where the update timing of the intermittent display mode is slowerthan that of the live-image display mode.

OTHER EMBODIMENTS

[0075] It is to be understood that the advantages of the presentinvention can also be achieved by supplying a program code of softwarefor implementing the functions of the above-described embodiment to acomputer (CPU, MPU, etc.) of an apparatus or system connected to devicesso as to make the devices operate for implementing the functions of theabove-described embodiment and making the devices operate according tothe program stored in the computer in the apparatus or system.

[0076] In that case, the program code itself of the software achievesthe functions of the above-described embodiment, and thus the programcode itself and means for supplying the program code to the computer,for example, a storage medium storing the program code constitute thepresent invention.

[0077] The storage medium for storing the program code may be, forexample, a floppy (registered trademark) disk, a hard disk, an opticaldisk, a magneto-optical disk, a CD-ROM, a magnetic tape, a non-volatilememory card, a ROM, etc.

[0078] Furthermore, not only by the computer executing the suppliedprogram code, but also by the program code functioning in associationwith an OS (operating system) running on the computer, or anotherapplication software or the like, the functions of the above-describedembodiment may be achieved. This program code can be used in anotherembodiment of the present invention.

[0079] Further, after the supplied program code is stored in a memory ofa function extension board of the computer or a function extension unitconnected to the computer, a CPU or the like in the function extensionboard or the function extension unit may execute part of or the entireprocess based on an instruction of the program code, thereby achievingthe functions of the above-described embodiment. This example is alsoone of embodiments of the present invention.

[0080] As described above, the present invention uses the live-imagedisplay mode for successively obtaining photographed images that aresuccessively input and the still-image display mode for obtaining aphotographed image that is input at predetermined time. Therefore,markers can be extracted by using photographed images obtained in eithermode. Further, the markers can be manually extracted from live images(this function is achieved by the function of maintaining the positionsof the extracted markers, even though the subject image is updated).Therefore, by switching between these modes, the accuracy of markerextractions is high under all conditions.

[0081] For example, where the surroundings constantly change or all thenecessary markers become alternately visible and invisible, then thelive-image display mode is selected, since all the necessary markerscannot be simultaneously photographed. Further, if it is difficult tocontinuously photograph the markers in good condition, then thestill-image display mode is selected. In this manner, the markers can beeasily extracted with a high degree of accuracy.

[0082] While the present invention has been described with reference towhat are presently considered to be the preferred embodiments, it is tobe understood that the invention is not limited to the disclosedembodiments. On the contrary, the invention is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

What is claimed is:
 1. An information processing method for obtainingcamera parameters of an image pickup device, the method comprising thesteps of: selecting at least one mode from among a plurality of obtainmodes including a first obtain mode for obtaining moving images, and asecond obtain mode for obtaining at least one still image; obtaining theimage from the selected mode; extracting markers included in the image,which is obtained through said image obtaining step; and determiningcamera parameters of said image pickup device by using image coordinatesof said markers extracted in said marker extracting step.
 2. Aninformation processing method according to claim 1, further comprisingdisplaying the determined camera parameters of said image pickup device.3. An information processing method according to claim 1, wherein thecamera parameters are determined such that the sum of differencesbetween theoretical and measured values of said markers is minimized. 4.An information processing method according to claim 1, wherein saidmarkers have known world coordinates and said determining step is usedfor deriving camera parameters of said image pickup device by using theimage coordinates and world coordinates of said markers.
 5. Aninformation processing method according to claim 1, wherein said markersinclude at least three markers.
 6. An information processing methodaccording to claim 1, further comprising the step of displaying andsuperimposing said markers extracted in said marker extracting step onsaid image.
 7. An information processing method according to claim 1,further comprising providing a first extraction mode for automaticallyextracting said markers and a second extraction mode for manuallyextracting said markers.
 8. An information processing method accordingto claim 7, wherein said first extraction mode automatically extractsmarkers based on predetermined extraction conditions.
 9. An informationprocessing method according to claim 8, further comprising the step ofchanging said extraction conditions before extraction is performed insaid marker extracting step.
 10. An information processing methodaccording to claim 1, further comprising the step of restrictingoperation of said determining step, wherein the number of said markersextracted in said marker extracting step is determined, and if thedetermined number is less than a predetermined number, the operation ofsaid determining step is restricted.
 11. An information processingmethod according to claim 1, further comprising the step of alerting auser, wherein the number of said markers extracted in said markerextracting step is determined, and if the number is less than apredetermined number, the user is alerted.
 12. An information processingmethod for obtaining camera parameters of an image pickup device whosecamera parameters are fixed, the method comprising the steps of:switching between a first display mode for displaying images picked upby said image pickup device, as moving images, and a second display modefor displaying at least one image picked up by said image pickup device,as a still picture; displaying an image picked up by said image pickupdevice in one of the first and second display modes, the display modebeing selected in said display-mode switching step; extracting markersincluded in the picked-up image, which is displayed in said imagedisplaying step; and determining camera parameters of said image pickupdevice by using image coordinates of said markers, which is extracted insaid marker extracting step.
 13. An information processing method forobtaining camera parameters of an image-pickup device whose cameraparameters are fixed, the method comprising the steps of: obtainingimages picked up by said image pickup device, as moving images;extracting markers included in at least one of the obtained images,which are obtained in said image obtaining step; and determining cameraparameters of said image pickup device by using image coordinates ofsaid markers, which is extracted in said marker extracting step.
 14. Aninformation processing apparatus for obtaining camera parameters of acamera whose camera parameters are fixed, the system comprising: a unitfor selecting at least one mode from among a plurality of obtain modesincluding a first obtain mode for obtaining images picked up by saidcamera, as moving images, and a second obtain mode for obtaining atleast one image picked up by said camera, as a still image; a unit forobtaining an image picked up by said camera in one of the obtain modes,the obtain mode being selected by said obtain-mode selecting unit; aunit for extracting markers included in the image, which is obtained bysaid image obtaining unit; and a unit for determining camera parametersof said camera by using image coordinates of said markers, which isextracted by said marker extraction unit.
 15. A computer program formaking a computer function as an image processing apparatus of claim 14.16. A computer readable storage medium storing a computer program ofclaim
 15. 17. An information processing method for obtaining cameraparameters of an image pickup device, the method comprising the stepsof: displaying, as moving images, images that are picked up by saidimage pickup device; determining image coordinates of markers bymanually using a pointing device to point out their positions on saiddisplayed moving images; and determining camera parameters of said imagepickup device by using the determined image coordinates of said markers.18. The method of claim 17, wherein the camera parameters are fixed. 19.The method of claim 1, wherein the camera parameters are about cameraposition and orientation.